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ABSTRACT 



This study examined the perceptions of basic education 
school teachers in Turkey to the educational software evaluation criteria, 
developed for the acquisition of multimedia instructional software by the 
Turkish Ministry of National Education, from the viewpoint of actual users. 
Data were collected through structured interviews, group discussions, and 
participant observation. The findings indicate that instructional adequacy is 
the most important criterion both from the theoretical and from the practical 
perspective; a second factor mentioned was that effective instructional 
software should include real-life examples and simulations of abstract 
concepts. Criteria considered as having secondary importance were the variety 
and number of exercises, the aspect of developing students' creativity and 
critical thinking capacity, presentation of accurate information, being free 
from scientific errors, individualizing learning, being in accordance with 
the pedagogical level of students, and being enriched through the use of a 
variety of sound effects, images, videos, and animations. Other questions 
addressed the conditions under which instructional software was used and 
issues to be taken into consideration in the future phases of instructional 
software procurement and technology assisted instruction. (Contains 11 
references . ) (MES) 
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Abstract; This stucb^ examines the perception of basic education school teachers on the 
theoretically developed educational software evaluation criteria fiom the viewpoint of actual 
users. Data was collected through structured interviews, groiq> discussions and partic^ant 
observatioa The findings indicate that instructional adequacy is the most inq>ortant criterion 
both fiom the theoretic^ and fiom the practical perspective. 



Background 

The age of technology carries with itself several requiremOTts for all educational institutions to respond 
in order to keep iq) with it. On the other hand, evaluation of educational technology programs has its place as a 
particularly hard task among others. As there is an increase in the number of countries malring efficient use of 
technology md developing technology plans and strategies to the aim of improving the quality of education and 
learning achievement, there rises the need for understanding the inqjact of technology on inqjroving the student 
achievement. 

The Ministry of National Education of the Republic of Turkey (MONE), like all other educational 
in^tutions, face the necessity of preparing students for the 21®* century. Towaixis this objective, the Ministry has 
initiated a reform program, first step of which has been to extend the period of basic education from five to eight 
years. This reform program was legalized through the passage of the Basic Education Law No.4306 in 1997, 
which also enacted several additional revenues to finance the Government s Basic Education Program. 

The Basic Education Program is the Government s action program to apply its new basic education 
strategy , the objectives of which are (a) to achieve universal coverage in an expanded eight-year basic 
education cycle, (b) to improve the quahty of basic education, and (c) to make basic education schools a learning 
soiHce for the co mmuni ty. The Program aims to achieve these objectives through (a) expanding the capacity of 
basic education schools throughout the country, (b) facilitating school attendance of children who are least likely 
to attend school, (c) reducing classroom overcrowding and double shifting, (d) improving training and incentives 
for teachers, (e) improving the si^ly of educational materials to basic education schools, (f) introducing 
con^uter-aided learning, and (g) increasing parental and community involvement in schools (World Bank, 
1998). 

Total estimated cost of this program is USD 1 1 .2 billion for a total of three years. The aforementioned 
revenues would finance ^proximately 90% of the Program, and to respond to the remaining, a partnership 
been established with the World Bank, which has mainly focused on the qualitative part of the Program. USD 
300 million loan was received fiom the World Bank to finance the first phase of the whole Basic Education 
Program, which consists of renovation, rehabilitation and construction of additional frtcilities for 340 basic 
education schools in rural and slum areas, provision of instructional materials, textbooks, and stationary to those 
students in rural schools, and to provision of 2828 IT Rooms in 2451 basic education schools throughout the 
countiy. These 2451 schools have been selected fiom all provinces on the basis of at least two sdKX)ls fiom each 
sub-province. The scope of the IT Component of the Basic Education Program will be expanded in the second 
and third phases to cover approximately 1 5, OCX) basic education schools located in Turicey. 

To the aim of improving the quahty of education, the Ministry of National Education has intended to 
provide all basic education schools with IT Rooms where children will become cornputer literate and work with 
computers to sv^rt their education (MONE, 1 999). Since the objective is to put the technology into the use in 
all basic education schools, the IT Rooms have been designed to have, besides computers, overhead projectors 
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and some other technologies. In order to enable an efficient and full use of these technological equipment by the 
studrats and teachers, the Ministry has also aimed at providing all instructional materials to these schools, 

ranging fiom transparencies, educational video tapes, handbooks for teachers, and multimedia instructional 
software. 

.^ongst the Ministry s endeavor to create an effective learning environment for teachers and students, 
the provisron of multimedia instructional software has had a very important place, since acquisition of software 
needs a very detailed examination and evaluation of the software submitted. The Ministry, as for the 
procurement of computer hardware and other technological equipment, has held an intPmgtinn^l* competitive 
btdding for the procurement of multimedia instructional software. The preparation of the bidding documents, 
Jocularly of the technical specifications, has been done through a hard process, on the basis of a thorough 
literature review. The technical specifications, prepared in coordination with the specialists, have consisted of a 
detailed evaluation criteria based on national and international literature and applications, to be used for the 
procurement of educational software including multimedia instructional software in several subject areas 
between 1 -8 grades (English, German, French, Turkish, Science, Social Sciences, and Mathematics) 
electronic references, and educational games. 

The criteria prepared by the MONE is based on four main areas; namely, (1) curriculum adequacy, (2) 
instructional adequacy, (3) programming adequacy, and (4) cosmetic adequacy. There are several items under 
each title, which are clearly defined and scored accordingly. The essential hems within the criteria have 
consisted of consistency with the curriculum, adequacy to the pedagogical level of the students, ease of use, use 
of colors, etc, among which the consistency with the curriculum has been the primary one: Any software which 
could not respond to this item would not be taken to further evaluation. 

Following the bid opening, the Ministry of National Education has established committees for technical 
evaluation of the multimedia instructional software submitted to the bidding. The committees consisted of 
subject area teachers who were currently working in the basic educational schools, subject area teachers who 
were currently working for the Board of Education of the MONE (a Board responsible for evaluating and 
approving any kind of instructional material to be utilized in primary and secondary education level), conqjuter 
teachem, Turkish language teachers, and pedagogues. The conrmittees have reviewed, evaluated, and scored 
accordi^ly a total of 230 instr^onal software within 6-month period. This period has also covered a fifteen- 
day basic training on multimedia instructional software evaluatioa Those teachers participated in the technical 
evaluation commit^s were, then, asked for evaluating the criteria on which the technical evaluation has been 
based, and for providing recoimnendations fiom the perspective of practicing teachers. 



The Study 

The purpose of this study is to explore the applicability of the instructional software evaluation criteria 
develop for the acquisition of Multimedia Instructional Software by the Turkish Ministry of National 
Educatioa The study focuses on what teachers who have actively participated in the evaluation process have 
found most practical/feasibW^licable and what have found unnecessary or to-be- modified among those 
criteria developed on the basis of literature and theory. 

For instructional software evalu^on, there has been an crescent need for qualified human resources 
required, towards the provision of an efficient le arning environment and the improvement of learning quality and 
achiev^ent This is also the case for Turk^. Besides, Turkey do not have a specific research institution 
specialized on software development or evaluation process. However, it is possible to obtain information fiom 
various researches on in-class use of technology (U.S. Department of Education, 1988) and other studies 
conducted by ^vate software evaluation con^anies (Children Software Review, 1995). Among the features of a 
good mstructional software, articles, dominantly, speaks of motivation, user control, Internet access, timely and 
efficient feedback, pedagogical adequacy, interactivity and instructional content (Harmefin & Pack, 1988; Alessi 
& Trollip, 1991; Comer & Geissler, 1998; Baumgartner & Payr, 1996; Persichitte, 1995; Caftori & Paprzcki, 
1997, Abramson, 1998). Notwithstanding, the researches in this field, also forewarn the researchers about the 
plication of a social science ^iproach to the evaluation of multimedia products (Batrmgartner & Payr, 1996; 
Reeves, 1993). Thus, it is essential to know how the literature-based educational software evaluation criteria 
differ ^m any other software evaluation criteria developed considering teaching experience and plication of 
utmost importance and what the impact of such criteria on output is. 

Instructional software evaluation criteria observed during the literature review have focused on the 
inqxjrtance of instructional quality, which is one of the most significant among the Ministry s evaluation criteria. 



The criteria observed have foreseen to support the instruction process through the use of several instructional 
strategies where computer use has the central role, via following such basic steps as stimuli, guidance, drill and 
practice, and assessment (Abramson, 1998). 

Other essential issues emphasized in the hterature, besides instructional content, have consisted of 
efficient and proper use of computers through the aim of related instructional objectives, being free of technical 
errors, and use of accurate engineering processes (Persichitte, 1995). 

Some researches conducted as to the instructional software evaluation have been based on applying a 
supervision mechanism to some extent for the use of instructional software. Another relevant issue has been that 
some instructional software labeled as instructional or educational may not be instructional or 
educational at all, and those software may not serve to instructional or educational objectives. Hence, these 
researches point out the problem of evaluation and supervision, two important criteria in the use of instructional 
software. This brings another question: the necessity for taking into consideration some other criteria to enable 
the teacher to evaluate and supervise student use and achievement, along with student control (Caftori & 
Paprzcki, 1997). 

Another problem taken into consideration in the researches concerned has been that the instructional 
software evaluation processes have not included professionals specialized on pedagogy. Similarly, those 
researches, which have revealed the problems encountered as to the instructional software evaluation methods, 
have recommended a social sciences approach to software evaluation process in order to overcome these 
problems (Baumgartner & Payr, 1996). 

In view of these studies and researches, this study lays open to view the mstructional software 
evaluation criteria prepared and utilized by the Ministry of National Education for the acquisition of off-fri^shelf 
instructional software, through making use of qualitative research method. Information has been obtained from a 
group of those teachers participated in software evaluation process through interviews and open-ended questions. 
The study has also en^loyed a variety of data collection methods ranging from documentary evidences to 
interpretation of the comments of the senior staff of the Ministry who are a part of the Ministry s education 
pohcy. Data analysis method used in this study consists of data examination, data reduction, data display, and 
finally conclusion drawing from specific to a more generalized (Miles & Huberman, 1994). 



Results 

As stated before, participants of this study are tiiose teachers participated in the Ministry s multimedia 
instructional software procurement evaluation process. 10 volunteer participants were observed from the 
beginning of the evaluation process. Besides these observations, 10 participant teachers were asked to respond, 
in written, to four open-ended questions at the end of the evaluatioa Responses to these questions have shown 
certain emphasis on certain issues as to the instructional software evaluation. 

Responses to the first question, \viiich is for determining the conditions, as a teacher, under which 
instructional software may be us^ have pointed out that necessary physical conditions and hardware. Three of 
the participants have stat^ that it is essential to provide a compute per student for an efficient in^lementation 
of technology assisted instructioa Merely two of the participants have enq)hasized a re-arrangement of the 
curriculum, and an effective tiine management for technology assisted instruction. Quahty of the instructional 
software, amount and variety of the instructional software to be used, and provision of in-service training to 
teachers on the use of instructional software to siqiport educational activities are the feictors disregarded by the 

participants. r j 

The second question is related to criteria, to which instructional software has to respoiid, to be utdi^ 
for educational purposes. Participants responses to ftiis question have concentrated on the critenon of bei^ 
consistent witii and siqiportive of the curriculum. The second fector mentioned has been that an effective 
instructional software should include examples from real life and simulations of abstract concepts in real life to 
allow the students these concepts with their actual fives. The criteria considered as having secondary importance 
have been the variety and number of exercises, aspect of developing students creativity and critical t hinking 
capacity, presentation of accurate information, being free fix>m scientific errors, individuafizmg lea r ni ng , being 
in accordance with the pedagogical level of students, and being enriched throu^ the use of a variety of sound 
effects, images, videos and antmatinns Content-related richness, motivation and technical features have only 
been emphasized by two participants. 

The shortcomings observed in the evaluation process of the instriictional software submitted for the 
Ministry s procurement process have included, primarily, consistency with the curriculum, being supportive to 
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tbe cimculin^ proper use of Turkish language, proper use of audio-visual effects in required amount and 
consistency with the pedagogical level of users. It has been stated that software s being free of technical errors 
I^vmt stunts disappomtment and losing their motivation. It has also been emphasized that informins 
studrats of the objectives before the lesson shall have a positive impact on learning. In view of the fact that the 
p^cipmts have not mentioned about the provision of examples from real life and the use of effective 
si^afrons, which have been emphasized for the second question, it has been considered that the software 
suDmitted to the Mimstry have responded well to these criteria. 

The la^question asked was related to the issues to be taken into consideration in the fiiture phases of 
insh^ond software procurement and technology assisted instruction. Two answers to this question were 
OTphasized muck The criterion of consistency with the pedagogical level of students has also been of utmost 
mqiortance for this questioa Another significant issue underlined by the participants has been the formal 
mea^s to be taken at the senior level and circulation or regulations to be issued by the Ministry. These 
^rhcipants have stated that the central Ministry should take necessary measures to ensure the efficient use of IT 
RTOms. Thrre of the participants have pointed out the necessity of strengthening local software production 
s^or, ^portag an mcrease m the number of software companies, and accordingly the number of software in 
the i^ket, and aisunng competition among the software market Issues having secondary importance have been 
stated as takmg into consideration national conditions for software production, and as preparing the technical 
specification m more detaU. Some participants have recommended as essential as having modem and 
co^porary content, ensuring interactivity with the student, and providing teacher training on the use of 
instructional software. Only one participant has emphasized the importance of having students views and 
co^ents durmg the software production and development process, since they are the actual users of the 

software Another participant underlined the necessity of having a pilot application for efficient use of such a 
large-scale project. ouvu a 

Talmg into consideration the socio-economic status of the participants and the schools where they were 
woikmg. It has been ob^ed that these participant teachers have been working in those schools attended by 
lower-mi^e-class children, and that, accordingly, they have not been femiliar with technological applications 
Hence, while participant teachers have been responding weU to the efforts for technology assisted instruction, 
according to them, the provision of physical conditions and necessary hardware are the primary criteria for using 
technolo^ m educ^onal activities since they have had no opportunity for application. In parallel, particii^ 
teachere have ^t^ that responsibility of expansion and of efficient implementation of computer literacy and 
technology assis^ instruction has been of the Ministry, and emphasized the importance of teacher training and 
necessary fonnal measures for effective use of technology in education. 

When we look at the Ministry s part, we see that the most essential criterion of an instructional software 
from the persp^ve of fte semor staff of the Ministry has been the instructional content. Consistency with the 
cumc^i^ and at least bemg supportive of the curriculum, is of utmost importance for the central Ministry. The 
second mportMt ^tor for the Ministry is the consistency with the national values of Turkish culture and 
presentmg no devi^on fiem the constitution of the Republic of Turkey. The written content and all other 
pictures and animation should also be respectful to these criteria. From the interviews, it has been seen 
that the ^or ktostiy s^ considers teachers as the most essential factors and, believes in the importance of 
^hCT traiMg for an efficient implementation of technology assisted education. The provision of physical 
^tira and n^essary hardware is of secondary importance to the Ministry because the Ministry assures that all 
physical conditions can be provided with no problem. 

Conclusion 



Tlose issues, received by the participant teachers both through observations and interviews, show no 
devi^on either from the lit^ture or from the criteria prepared by the Ministry based on the literature. As 
stated in the ht^ture, the participant teachers have emphasized most the importance of preventing the 
instructional and educational nature of the instructional software, of being siqiportive to the cuiriculv^of 
CT^g mteia^vity with the student, of motivating students, of reinforcing the students through the use of 
drills and exercises, and of having simulations to present abstract concepts to students. 
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